A Hypergraph-Based Workload Partitioning Strategy for Parallel Data Aggregation

This paper presents an algorithm to e ciently carry out data aggregation operations on large disk-based datasets on a parallel machine. This algorithm employs a hypergraph formulation for partitioning the workload among processors. Data aggregation is a common operation executed by applications that explore and analyze very large multi-dimensional scienti c datasets. A data element in these datasets is associated with coordinates in a multi-dimensional attribute space. Depending on the application area, the datasets may include data produced by scienti c simulations and experimental measurements obtained from sensors attached to satellites and microscopes, and from seismic data and tomographic imaging techniques. In many cases, analysis is performed by (1) selecting a subset of the dataset, (2) mapping input data elements to output data elements, and (3) computing or updating the values of an output element by aggregating the values of the input elements that map to the same output element. Mapping between input and output elements is often based on the coordinates in the underlying attribute space(s). Note that an input element may map to multiple output elements, and an output element may be mapped to by multiple input elements. Typical aggre-